Targeting a light-weight and multi-channel approach for distributed stream processing

Processing high-throughput data-streams has become a major challenge in areas such as real-time event monitoring, complex dataflow processing, and big data analytics. While there been tremendous progress distributed stream processing systems the past few years, low-latency (a.k.a. high sustainable-throughput ) requirement of modern applications is pushing limits traditional infrastructures. This paper introduces new engine (DSPE), called Asynchronous Iterative Routing (or simply “AIR”), which implements light-weight, dynamic sharding protocol. AIR expedites direct asynchronous communication among all worker nodes via channel-like protocol on top Message Passing Interface (MPI), thereby completely avoiding need for dedicated driver node. The system adopts progress-tracking protocol, hew-meld , experimentally observed to show low latency our master-less architecture when compared conventional low-watermark technique. current version also equipped with two fault tolerance recovery strategies namely checkpointing & rollback replication . With its unique design, scales out particularly well multi-core HPC architectures; specifically, we deployed it clusters up 16 448 cores (thus reaching peak 435.3 million events 55.14 GB processed per second), found significantly outperform existing DSPEs. • A open source architectures. Specifically,